Add safe context watcher API by florentinl · Pull Request #6227 · PyO3/pyo3

florentinl · 2026-07-23T11:16:27Z

Adds PyContext and safe context-watcher bindings for GIL-enabled CPython 3.14 and newer:

typed context-switch events
bound and unbound RAII watcher guards
panic- and error-safe callback trampoline

PyContext itself is available on all supported CPython versions.

use pyo3::{
    prelude::*,
    types::{
        context::{watch_callback, ContextEvent},
        PyContext,
    },
};

fn context_changed(_py: Python<'_>, event: ContextEvent<'_, '_>) -> PyResult<()> {
    if let ContextEvent::Switched(context) = event {
        println!("current context: {context:?}");
    }
    Ok(())
}

fn main() -> PyResult<()> {
    Python::initialize();

    Python::attach(|py| {
        let _watcher = PyContext::add_watcher(py, watch_callback!(context_changed))?;
        py.run(
            c"import contextvars\ncontextvars.Context().run(lambda: None)",
            None,
            None,
        )
    })
}

Tested with unit tests, doctests, and Clippy on Python 3.14.

chirizxc · 2026-07-28T13:21:33Z

LGTM, but im not a maintainer

florentinl · 2026-07-28T13:35:00Z

Still, thank you for taking a look.

## Description Add a generic CPython 3.14+ publisher for the internal `python.context.switch` event. It registers the CPython `PyContext_AddWatcher` API from the native extension and dispatches the event after each context switch. The C callback catches Rust panics at the FFI boundary and preserves any exception already pending when CPython invokes it. This PR is stacked on #19604, which adds the public `DD_TRACE_OTEL_CTX_ENABLED` configuration and activation-listener setup. The setting defaults to `true`; setting it to `false` disables both the listener and this Python 3.14 watcher. It also complements #19336, which independently provides the asyncio, uvloop, AnyIO, and greenlet emitters for older Python versions. ## Testing - `scripts/lint checks` - `reno lint` - CPython 3.14 native context watcher: 4 passed - Linux CPython 3.14.5 OTel thread context: 6 passed ## Risks Low ## Additional Notes - The PR contains raw external C bindings to CPython. I am currently upstreaming them to PyO3; once that work is merged and released, we can rely on the PyO3 bindings and simplify this code: - PyO3/pyo3#6204 - PyO3/pyo3#6227 Co-authored-by: florentin.labelle <florentin.labelle@datadoghq.com>

codspeed-hq · 2026-08-20T13:44:01Z

Merging this PR will not alter performance

✅ 141 untouched benchmarks
⏩ 6 skipped benchmarks¹

_{Comparing florentinl:florentin.labelle/add-safe-wrappers-for-context-watching-api (1f0fe5a) with main (ac9b689)}

6 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

davidhewitt · 2026-08-22T18:14:36Z

I would quite like to avoid the macro for this API, it seems unfortunate. I understand why it's there, however. I'm thinking about what we can do as alternatives.

davidhewitt

I opened florentinl#1 which has a suggestion how I'd like this API to look.

Also placed a few other comments here.

florentinl · 2026-08-27T15:17:42Z

Thank you a lot @davidhewitt for the thorough review and your design suggestion.

I updated the PR to remove the macro API in favor of the design you suggested and tried to address your other comments.

davidhewitt

Thanks, this generally looks ready to me, just some small nits.

chatgpt-codex-connector · 2026-08-31T12:32:11Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T11:42:12.421778Z	`9def052`	New commits

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1a4afcdbe6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Fix the newsfragment's overclaim that PyContext requires GIL-enabled CPython 3.14+ (it's available on all supported versions; only the watcher API needs 3.14+), clarify the SAFETY comments on ContextWatcherGuard around single-interpreter attachment, and drop a redundant doctest cfg attribute already covered by the module-level gate in lib.rs.

Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9def05278d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-09-02T11:42:09Z

+    //   attaching to more than one interpreter, so this is the interpreter for which the watcher
+    //   was registered
+    // - `watcher_id` was returned by `PyContext_AddWatcher`
+    error_on_minusone(py, unsafe { ffi::PyContext_ClearWatcher(watcher_id) })


Defer clearing watchers during callback dispatch

On GIL-enabled CPython 3.14.4, if an earlier watcher uses safe shared state to call clear on a later watcher, this immediately nulls the later callback slot while CPython is still iterating a snapshot of the active-watcher bitmask; CPython subsequently calls that null slot and segfaults. The drop path has the same issue because it also calls PyContext_ClearWatcher immediately, so the safe API permits memory-unsafe behavior. Track callback dispatch and defer removals until it completes (or otherwise prevent clearing another watcher from within a callback).

Useful? React with 👍 / 👎.

I don't really know how it should be handled in PyO3.
python/cpython#155943 which was opened to fix python/cpython#155619 solves the problem.

For now I think we can call this a CPython bug and merge this as-is in PyO3; it's a silly edge case which I think we can allow to crash even if it's not safe.

davidhewitt · 2026-09-06T18:12:51Z

Looks like something still not properly skipped in the wasm tests.

chirizxc reviewed Jul 24, 2026

View reviewed changes

Comment thread src/context.rs Outdated

chirizxc reviewed Jul 24, 2026

View reviewed changes

Comment thread src/context.rs Outdated

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from eb0dff3 to 66ad857 Compare July 28, 2026 07:47

florentinl marked this pull request as ready for review July 28, 2026 08:19

florentinl requested a review from chirizxc July 28, 2026 08:19

chirizxc reviewed Jul 28, 2026

View reviewed changes

Comment thread src/context.rs Outdated

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from 7259b06 to 69b141c Compare July 28, 2026 10:40

florentinl requested a review from chirizxc July 28, 2026 11:20

florentinl requested a review from bschoenmaeckers July 29, 2026 15:19

florentinl mentioned this pull request Aug 3, 2026

feat(tracing): publish Python context switches natively DataDog/dd-trace-py#19431

Merged

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch 2 times, most recently from b253b69 to 2041732 Compare August 13, 2026 13:27

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch 2 times, most recently from d0a8de6 to cf95e72 Compare August 20, 2026 13:32

davidhewitt reviewed Aug 22, 2026

View reviewed changes

Comment thread src/context.rs Outdated

Comment thread src/context.rs Outdated

Comment thread src/context.rs Outdated

Comment thread src/context.rs Outdated

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from cf95e72 to e4ac1c4 Compare August 27, 2026 12:24

florentinl requested a review from davidhewitt August 27, 2026 15:17

davidhewitt reviewed Aug 29, 2026

View reviewed changes

Comment thread src/types/context.rs Outdated

Comment thread src/types/context.rs

davidhewitt reviewed Aug 29, 2026

View reviewed changes

Comment thread src/types/context.rs Outdated

chatgpt-codex-connector Bot reviewed Aug 31, 2026

View reviewed changes

Comment thread src/types/context.rs

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch 2 times, most recently from a9e5201 to c0f278b Compare August 31, 2026 13:35

florentinl requested a review from davidhewitt September 1, 2026 11:24

davidhewitt reviewed Sep 2, 2026

View reviewed changes

Comment thread src/types/context.rs

florentinl and others added 21 commits September 2, 2026 13:35

Add safe context watcher API

e128235

Add newsfragment

fc02fb9

Refine context API availability

c6f424b

Guard macro-dependent context tests

3f70777

Refine context watcher safety and tests

d238aee

add forgotten guard test

694a3a9

Isolate forgotten context watcher test

dafb6af

Simplify context watcher event conversion

b5d47c4

add context watcher guard as associated method

2a9397e

Use checked context watcher event conversion

7926e54

Remove watcher-specific trampoline error check

b09e949

Support unbound context watcher guards

ac88a33

Fix race in unbound context watcher test

c5636e5

Fix race in unbound context watcher clear test

c0e49d7

Refactor context watcher tests

8edd88a

Apply suggestions from code review

5e5eb07

Co-authored-by: David Hewitt <1939362+davidhewitt@users.noreply.github.com>

Reduce context watcher trampoline code size

f623224

Handle watcher callbacks which leave exceptions set

e7eca08

Use no-std mutex helper in context watcher tests

349d904

Skip threaded context watcher test on wasm

9def052

florentinl force-pushed the florentin.labelle/add-safe-wrappers-for-context-watching-api branch from c0f278b to 9def052 Compare September 2, 2026 11:38

chatgpt-codex-connector Bot reviewed Sep 2, 2026

View reviewed changes

Skip watcher panic test without std

1f0fe5a

davidhewitt approved these changes Sep 6, 2026

View reviewed changes

davidhewitt added this pull request to the merge queue Sep 6, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 6, 2026

davidhewitt added the CI-build-full label Sep 6, 2026

Uh oh!

Conversation

florentinl commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chirizxc commented Jul 28, 2026

Uh oh!

florentinl commented Jul 28, 2026

Uh oh!

codspeed-hq Bot commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will not alter performance

Footnotes

Uh oh!

davidhewitt commented Aug 22, 2026

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

florentinl commented Aug 27, 2026

Uh oh!

davidhewitt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

florentinl Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

davidhewitt Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

davidhewitt commented Sep 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

florentinl commented Jul 23, 2026 •

edited

Loading

codspeed-hq Bot commented Aug 20, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Aug 31, 2026 •

edited

Loading